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ABSTRACT 

A stakeholder approach was used in the evaluation of 
the basic science courses of the University of North Dakota School of 
Medicine. The stakeholder approach attempted to structure the working 
relationship between the evaluator and the program stakeholders so 
that they participated in choosing issues r data to be collected , and 
ways to organize and present the data. The clients in this evaluation 
had a sense of input and control; changes accepted as policy 
directives were based on their recommendations. Growth in a planned 
direction resulted from this evaluation process. The model used, one 
of evaluation as an active process occurring in a political 
situation r is adaptable to many situations. Factors contributing lo a 
successful stakeholder evaluation include: (1) identifying the 
decision makers; (2) defining the evaluation's nature, character, and 
purpose; (3) establishing congruence between the goals and objectives 
of the evaluator and the audience; (4) legitimizing different stakes 
in the program; and (5) providing information to the appropriate 
person in an acceptable format. (SLD) 
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A Curriculum Evaluation Using the Stakeholder 
Approach as a Change Strategy 

Evaluations may serve multiple functions : clarifying goals and 
objectives, determining criteria for measuring success, 
identifying unintended outcomes, assessing the value of a Program 
- but ultimately, the purpose of an evaluation is to P^ X J? 
information for making decisions. In the 1970s ^utilization of 
evaluation results became a major research concern as evaiuators 
began to realize that evaluation results were not contributing to 
concrete decisions (Hueftle, 1984). There has been increasing 
recognition that even methodologically sound evaluation researcn 
is not used either by those being evaluated or by decision makers 
(Goldstein, Marcus, & Rarsch, 1978; O'Reilly, 1981) . One 
evaluator summarized the problem with this lament: With all trie 
money, time, effort, skill, and iritation that went into the 
acquisition of information, why does it generally have so little 
impact?" (Weiss, 1972, p. 26). Consequently, a number of 
evaiuators have suggested selecting evaluation protocols 
according to their potential for providing practical and usetui 
results (Patton, 1982; Rossi & Freeman, 1982; Rovers, 1986). 

Stakeholder, or responsive evaluation, evolved in the late 
1970s at the National Institute of Education due to an increasing 
concern with the lack of utility of t valuation results (Weiss, 
1983). It was hypothesized that attempts to rely on quan titative 
methods, to remain apolitical and to organize evai«at*ea 
evaluations around issues that should exist result in obtaining 
ineppropriate information. Data remains unused because it 
relates ■ o questions which are not of primary concern to persons 
involved in the program and hence, evaluation results do not have 
an impact on program change. Basically, the stakeholder approach 
attempts to restructure the working relationship between tne 
consultant and groups involved in the program, the stakeholders, 
in such a way that the stakeholders participate in deciding the 
issues that the evaluation will address, the kirds of data to be 
collected, and how it will be organized and presented. 

At M ie University of North Dakota School of Medicine, the 
curriculum phases are evaluated or. a rotating three year basis. 
Through the Office of Medical Education, I was asked to assist 
with the evaluation of Phase II, the basic sciences, and I 
utilized a stakeholder approach: Information would be provided 
only if there was inteaded usage and a decision making purpose, 
i.e. , a felt need. 



METHOD 



Gold (1981) outlines the essential features of the stakeholder 
approach which include assessing initial program capability 
identifying stakeholders and determining their expectations, 
matching stakeholder expectations with program capabilities; and 
timely, regular feedback to stakeholders of evaluative 
information as the program is implemented. The model does not 
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prescribe a strictly defined set of procedures; gather it 
suggests a cooperataive process in which the clients interests 
are foremost. This client orientation influenced planning at 
each s*- j.ge of the evaluation. 

Determination of initial expectations began informally. I 
met with the administration, faculty, and students to 9 ax J}.«J 
sense of their expectations for the evaluation, i.e., what did 
they want and need to know? Specific concerns were elicited. 
These ranged from very broad (Are students being prepared to 
begin clinical training?) to the very specific (How many hours 
did you study last night? ) . A working group which included 
representatives of the students and faculty, course directors, 
and two deans rated the potential questions and selected those to 
be addressed, thus limiting the scope of the evaluation. At 
meetings with this group. I described the stakeholder approach 
and asked about such requirements as deadlines, the best way to 
present the data, who should receive feedback, etc. 

The proposed design emphasized obtaining descriptive data 
rather than prescribing any manipulation of the program. I 
suggested multiple methods for data-gathering so that the issues 
would be covered from various perspectives and provide a 
crosscheck on the information received. The working group 
elected to collect data through small group discussion, a 
checklist, and interviews. Finalizing the design was a 

cooperative process relying on the stakeholder group for input 
and problem-solving, e.g., would they accept the information 
derived from student interviews as valid? Would students have 
the time to participate with finals approaching? After 
discussion, several volunteers developed and piloted the 
checklist and interview questions. An acceptable design was 
defined as one whose components reflected program intents, 
obiectives, and concerns of the audience. The design was not 
finalized and approved until it was acceptable to the entire 
group . 

Stakeholders participated in collecting the information: 
Course directors met with their faculty in small groups to 
discuss issues and report their findings, and students were 
interviewed. In both cases, the likelihood of predetermined 
responses was minimized, that is, interviews were relatively 
unstructured, and new questions were added as relevant concerns 
became apparent. Thus, the overall design continuously evolved. 
There was no attempt to control the setting, instead the goal was 
understanding through direct personal contact and experience with 
the program. More than one source provided information on each 
topic, but there was a determined effort to answer the questions 
and not provide more information than was needed to address the 
issues. 

Analysis of the data was organized around the stakeholders' 
concerns with an emphasis on collating themes or pictures from 
different sources. Both faculty and student perspectives were 
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related to each issue. Data included quantitative comments as 
applicable. Frequent informal dialogue and progress reports 
constituted early reporting. More formal documentation was 
organized around the stakeholders' questions; thus , students , 
individual faculty, the committee of representative stakeholders, 
and the administration required different information. 
Experiences were portrayed in these reports using the language of 
informants (for example, one topic was characterized as Mickey 
Mouse" ) , and descriptive material was used to provide a feeling 
of what the program was like. After discussion, recommendations 
were generated and voted upon. These passed through the 
administrative committee structure and became part of school 
policy. 

The stakeholder approach involved both a theoretical 
reorientation and a practical working strategy. It felt more 
active: I was not an impartial observer but a participant. It 
emphasized taking different perspectives and recognizing the 
complexity of the educational enterprise which has differing 
priorities, "stakes," at work in the process. Some frequent 
evaluation criticisms did not surface (Weiss, 1983). That is, xt 
wasn't irrelevant because it related to expressed needs; 
unrealistic, because no predetermined standards were set; or 
unfair, because the needs of less powerful groups such as 
students and supporting faculty were considered. 

Feedback from the first evaluation was generally positive, 
but I could only hope that the approach would be followed by 
actual change. As one of the faculty participants noted: I 
liked the evaluation. It was very well done, but I have- a 
problem with evaluations in general . I don t see anything 
changing. What good is the effort if it is impossible to make 
changes?" The succeeding evaluation three years later allowed me 
to determine the degree to which the evaluation had specifically 
addressed the problem of underutilization: Had the results 
influenced policy or program implementation? 

RESULTS 

Stakes (1975) defines the ultimate test of validity for an 
evaluation as an increase in the audience's understanding of the 
entity that was evaluated. A useful operational measure for this 
increase is a reduction in the number and level of concerns held 
by the audience and in the resolution of issues passed by them. 
Did this happen? Seven areas were targeted as needing 
improvement in the original evaluation. Briefly, they were: 
increasing clinical input, introducing community care needs, 
health care costs, and preventive medicine; practicing 
alternative forms of curriculum delivery; maximizing student 
self-learning; providing work-time estimates; concern with 
students' attitudes developing as a means of coping with 
pressure, i.e., becoming "less human"; and developing methods of 
assessing problem-solving skills. Six of the fl £ a l 

recommendations were related to concerns which had been suggested 
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as issues for the evaluation by the administration and faculty. 
One, attitudes and coping with stress, was an issue raised by 
students. To ascertain whether decisions had been made and 
implemented in the three years, program descriptions were 
obtained as before. A random sample of students one quarter ot 
tho class stratified by grade point average was interviewed using 
identical questions by the same interviewer. Faculty met with 
their respective course directors and completed the same 
questionnaire. Only one course director changed during this tim *r 
period and there were new faculty members. Questions which had 
arisen because of new concerns and/or requests for specific 
information were added on a separate sheet. Thus, althoughdata 
was being gathered to measure changes which had occured in 
response to the initial evaluation, the second evaluation 
incorporated new concerns so that it remained responsive to the 
current stakeholders . 

Content Changes 

The faculty indicate that clinical input increased dramatically. 
Of eight courses in the first year, the number of courses 
routinely presenting clinical correlations increased from one to 
four; using clinical speakers increased from four to six. Nev; 
methods of presenting clinical material such as field trips to 
the Rehabilitation Hospital and Veterans Adranistration occurred 
in three classes. Two courses added additional textbooks to 
provide case studies and clinical correlations. 

Community care needs had been introduced in one course, discussed 
in one and considered not applicable in six. Currently, these 
topics are introduced in four courses and discussed in two. 
Preventive medicine was added to three courses and a discussion 
of health care costs in one. 

Students seconded the faculty's opinions: During the 
student interviews, clinical input was recognized. They noted 
new experiences, e.g., interviewing real or simulated patients 
and field trips. In other cases, the type of input was the same, 
but it increased from "only minimal, tidbits" to "good." 

Instructional Delivery 

Additional self-learning materials are available in the form of 
self -instructional packages (available in five instead of four 
courses), course handouts (eight instead of six), and computer- 
assisted instruction (available in two rather than one course) . 
Instructors reported working on methods for student self- 
assessment. Students also noted an increase in the availability 
of learning aids in three classes and reported an increase in 
their use. For example, previously five Anatomy students of the 
thirteen interviewed were not sure if there were any out-of-class 
aids for self -learning. Audiovisuals were reportedly a low 
priority both in and out of class, and more handouts were 
requested. This year eleven of the thirteen students knew about 
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the availability of self -study aids, and nine actually used the 
videotapes. The number of learning aids increased from a 
reoprted zero in one course to four. All of the students were 
aware of the addition of. a computer test bank. 

The number of lecture hours was reduced Tor/ approximately 
15%. A total of 1050 is allowed (35 weeks at 30 hours of contact 
per week) and approximately 900 are utilized. Faculty now report 
using only slightly more lectures than other types of 
instruction, 488 hours compared to 401 hours of non-lecture (lab, 
discussion groups, problem-based learning, review sessions, 
etc. ) . 

Providing work- time estimates was a recommendation that did 
not substantively change or have any policy response. Although 
one more course reports having estimates, there is s* 11 ]- 1 
confusion about what they are and how they should be used. 
However, in response to concerns about the workload, tne 
administration initiated a stress program for freshman students 
in which they meet in small groups with sophomore facilitators to 
discuss concerns. 

Six of the thirteen students initially interviewed had 
characterized the volume of work as overwhelming. Although most 
experienced no real problems, i.e., actual failure, they felt 
they were pushed to their limits. The change this year is 
striking: No one reported being overwhelmed. Seven stated that 
the year was "not that bad"; it "could be managed." Students 
expressed an awareness that their attitude could make a 
difference. 

Developing methods of assessing problem-solving skills 

All courses now formally assess clinical problem-solving whereas 
before one-fourth of the courses thought this was not applicable 
in the basic science years. Over half of the courses also report 
an increased effort to assess clinical problem-solving skills 
informally, e.g., in discussions and lab. 

Other Changes 

Six of the seven recommendations were addressed. What about 
other developments during this period? Perceived changes that 
did not specifically address the recommendations included 1) less 
excessive demands on faculty time, and 2) evaluation procedures 
which received better ratings in terms of being "fair. In ten 
other areas which were reassessed such as integration of courses, 
teaching packages, and remediation, no change was noted. 
Student-initiated issues including lecture quality, desire for 
more flexibility, and the availability of self -learning also 
revealed no consistent change or new trends . 

Thus, the changes which occurred were based almost 
exclusively on program concerns and information described in the 
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first evaluation. Changes were validated independently by both 
faculty and students in both guantitataive (e . g. , an overall 
increaL in numbers) and qualitative (e g "^^nS XLlS 
fantastsic") terms. Areas not targeted did not chang r. ^culty 
initiated all of the changes except one under ^he direction and 
control of the Dean's office (the stress program) .The seven 
recommendations were approved by school committees , hence policy 
decisions were made. Implementation to varying degrees of six 
recommendations occurred. 



DISCUSSION 



What was the relationship of the evaluation to the changes that 
occurred; did the stakeholder approach make a ^"efjnce. 
According tc interviews with the Associate Dean and members of 
the administrative policy committees, the key fact °* f s °ne °f 
ownership. The clients had a real sense of input ^control. 
They determined the focus, gathered data, .^cussed and 
formulated recommendations. Their recommendations then went 
through official channels and were accepted as policy directives. 

Faculty members felt that the recommendations were 
implemented because the changes suggested were needed, group 
consensus existed; the recommendations were P^ a £f ^ °road 
directives rather than specific mandates which would be rejectea, 
they had a choice in how to implement the recommendations e . g., 
clinical input for some meant having a clinician speak, tor 
others, it meant using case histories. 

However, extradepartmental evaluations wer e generally 
perceived as a necessary evil. Faculty noted that teaching is 
only one responsibility and that keeping up with new content is 
mori important than how they are teaching. Most of the 
recommendations were piTHeived to be in the area J^l^ery *nd 
hence, had a low priority. It was admitted that changes in this 
area would be unlikely to occur without outside impetus. 

The evaluation also stimulated changes through informal 
mechanisms, i.e., it promoted discussion. Discussion at the 
corriDletion of an evaluation tends to center around tne 
recommendations which generates controversy and new ideas. 
Sometimes group pressure builds for change ma Particul ar a rea, 
for example, when the testing criteria in on J °° u "* " ™! 
equivalent to grading in other courses. When there is no group 
consensus, the process of change may be slower ^ a ™2 
dialogue has started. In Stake's terms, there was an ^crease m 
understanding as reflected by focusing on common concerns. 

There are, of course, multiple factors ^Y^^^Illv 
change. For example, having a new course director typically 
JeaSI to reorganization and an opportunity to implement changes^ 
It is naive to assume that no information discussion, ^planning 
existed prior to the evaluation findings Rather, evaluation 
results should be viewed as an additional source of information 
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which hopefully has an effect, if only in part, on decision- 
making. It is but one piece of information that flows into tme 
slow, developing process of program development (Alkm, 1975). 
Also, ideas may already be incubating, but it may take an 
external impetus to begin action. As one faculty member noted, 
"It was time." 

CONCLUSIONS 

Too often, evaluations are not used. An examination of the 
evaluation literature finds very few instances where the findings 
of an evaluation were adopted and led to a set of concurrent 
program decisions (Alkin, 1975). Despite good design and data, 
they seem to have no effect upon decision-makers or program 
outcomes. Yet, the primary rationale for an evaluation's that 
it provides information for action and contributes to decision 
making. "In short, in an evaluation it is important if someone 
needs the evaluation and someone cares" (Alkin, 1975, p. 201). 

There are universal characteristics of evaluation models, 
such as clarifying objectives and defining the role of the 
evaluator, which are applicable in any curriculum evaluation. 
However, selection of the features to be evaluated depends upon 
conceptualization of the philosophy and goals of the program. 
The stakeholder model, used in this evaluation, suggests that the 
information asked and data provided must be in response to 
concerns of people actively involved in the program if the 
evaluation is to have an impact. The presentation of data 
unrelated to actual concerns is unlikely to affect decision 
making. This approach proposes orienting directly to program 
activities rather than to program intents, responding to audience 
requirements for information, and allowing for different 
perspectives when reporting the results. In this case, issues 
discussed in the initial evaluation of Phase I led to the 
adoption of recommendations by the stakeholder groups. These 
recommendations became administrative policy and were returned to 
the group for implementation. Without further directives, 
changes evolved over the subsequent three years, changes apparent 
to both faculty and students. There were changes in all courses: 
changes in content, teaching, and evaluation. Overall, there 
seemed to be an orientation towards implementing the 
recommendations bv fitting them into the existing course 
structure. The method of initiating change and the type of 
change varied, but the end result was growth in a planned 
direction . 

Factors which contribute to a successful stakeholder 
evaluation are a) identifying the decision makers; b) defining 
the nature, character, and purpose of the evaluation; c) 
establishing congruence between the goals and objectives examined 
by the evaluator and those of interest to the audience; d) 
legitimizing different stakes in the program; and e) providing 
information as needed to the appropriate person in a format that 
will be acceptable. These activities are congruent with several 
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principles regarding successful implementation of chan 9 e ?! 
?nvSlvement of all who will be affected by the decisions to be 
made, a broad base of participation, and open lines 
communication (Mauksch & Miller, 1981). 

Potential problems in conducting a stakehlder evaluation 
include the amount of preliminary work which " required to 
determine initial expectations, conflicting issues between 
stakeholders who have no agenda or who do not want to know 
anytMng, or conversely, too many predetermined issues to ge 
cover, i 3 The approach assumes that it is P?"*^!*^ 
responsible and responsive to many issues even if this takes tne 
form of recognizing the political nature of th "* a ™^ 
process and legitimizing the diversity of interest The Process 
may be helped by involving representatives of all vested groups, 
limiting expectations, using multiple sources of J^^^ion, 
ascertaining that the data will have validity f ° r ^the clients 
and engaging in frequent dialogue with those in and around the 
program. 

The model used is one which is adaptable to many situations: 
Evaluation is perceived as an active process occurring in a 
political situation. There is no single truth to be unco vered 
on the contrary, a developmental process is being observed Each 
program affects many groups and their concerns are legitimate 
Frequently there is no follow-up to evaluations to determine if 
prior recommendations have been implemented. Using a stakeholder 
approach with repeated measures enables evaluators to plot and 
progress toward attainment of goals as well as to monitor changes 
in concerns and issues. 
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